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Effects-based  Decision  Analysis  Methodology  (EDAM): 
Designing  Revolutionary  Command  and  Control 

Systems 


ABSTRACT 

Traditional  cognitive  engineering  approaches  do  not  adequately  address  the  breadth  of  human 
performance  issues  in  revolutionary  Command  and  Control  (C2)  systems.  Consequently,  a  best- 
practices  approach  has  been  developed.  This  paper  describes  an  integrated  cognitive  engineering 
approach — the  Effects-based  Decision  Analysis  Methodology  (EDAM) — for  the  requirements 
analysis  and  design  of  revolutionary  command  and  control  systems  and  domains.  This  hybrid 
approach  uses  knowledge  elicitation  and  representation  techniques  from  several  current  cognitive 
engineering  methodologies.  The  techniques  were  chosen  to  allow  for  decision  analysis  in  the 
absence  of  an  existing  similar  system  or  domain.  EDAM  focuses  on  the  likely  system  or  domain 
constraints  and  the  decisions  required  within  that  structure  independent  of  technology,  existing  or 
planned. 

INTRODUCTION 

Every  new  system  design  falls  along  a  continuum  of  change,  from  minor  change  to  evolutionary 
change  to  revolutionary  change.  As  shown  in  FIGURE  1,  minor  change  involves  alterations  to  a 
system’ s  front-end  or  human  interface  but  no  substantial  change  to  the  back-end  system  and  no 
change  to  the  existing  work  organization.  Evolutionary  change  involves  substantial  alterations  to 
both  the  front-  and  back-end  of  the  system  but  no  substantial  change  to  the  work  organization. 
Revolutionary  change  involves  substantial  alterations  to  the  work  organization  which  necessitates 
entirely  new  front-  and  back-end  system  components.  Regardless  of  the  amount  of  change,  any 
design  effort  benefits  tremendously  from  a  thorough  needs  analysis,  which  includes  human 
performance  assessments  and  helps  develop  system  requirements. 

Many  methods  exist  that  can  determine  the  requirements  for  systems  undergoing  minor  changes. 
These  methods  focus  on  understanding  the  current  tasks  and  work  processes  in  order  to  improve 
the  operator  interface  to  support  those  tasks.  A  few  methods  were  developed  and  are  promoted 
for  evolutionary  change.  These  methods  focus  on  the  goals,  decisions,  and  constraints  of  the 
existing  system  or  domain  in  order  to  understand  how  best  to  control  and/or  interact  with  the 
existing  system  or  domain.  No  widespread  methods  exist,  however,  that  address  human 
performance  in  complex  systems  undergoing  revolutionary  change.  Developing  such  methods  for 
revolutionary  systems  is  difficult  because  no  current  work  organization  exists  for  analysis.  As  a 
result,  designing  new  revolutionary  systems  is  also  difficult. 

As  the  Department  of  Defense  moves  towards  a  net-centric  paradigm,  new  revolutionary 
command  and  control  (C2)  systems  and  work  organizations  are  necessary.  In  order  to  design 
systems  for  and  support  development  of  this  net-centric  framework,  JHU/APL  has  developed  an 
integrated  cognitive  engineering  approach.  Current  individual  cognitive  engineering 
methodologies  only  partially  capture  the  multiple  factors  affecting  humans  and  their  performance 


in  C2  work  organizations  and  primarily  focus  on  minor  change  and  evolutionary  designs,  not 
revolutionary  designs.  The  JHU/APL  integrated  methodology  considers  the  operating 
environment,  collaboration  between  humans  and  machines,  user  goals,  operator  decision  methods 
and  styles,  battle  rhythm  dynamics,  multi-level  security,  cognitive  strengths  and  limitations  and 
physiological  factors,  such  as  stress  and  fatigue,  for  a  revolutionary  new  design. 

The  Effects-based  Decision  Analysis 
Methodology  (EDAM)  employs  various 
knowledge  elicitation  and  representation 
techniques  from  current  cognitive  engineering 
methodologies.  EDAM  is  intended  to  be  used 
throughout  the  design  and  development  of  a 
prototype.  Information  gathered  with  EDAM 
will  also  be  used  throughout  the  project  to 
evaluate  human  performance  in  the  proposed 
system.  Although  EDAM  was  developed  to  aid 
in  the  design  of  revolutionary  systems,  it  may  be 
applied  to  systems  undergoing  minor  and 
evolutionary  changes. 

COGNITIVE  ENGINEERING 
AND  EDAM 

The  goal  of  cognitive  engineering  is  to  provide 
optimal  interoperability  between  human 
operators  and  today’s  complex  systems  so  that 
human  operators  can  more  effectively  perform 
their  duties  and  overall  system  performance  improves.  This  goal  is  particularly  important  for 
Command  and  Control  (C2)  design  where  warfighters  use  information  from  various  sources  to 
make  critical  decisions  in  the  planning  and  execution  of  strategic,  operational,  and  tactical  goals. 
Understanding  user  goals  and  decisions  is  critical  in  ensuring  that  the  total  system  provides 
utility. 

In  addition  to  the  goals  (i.e.,  desired  effects)  of  warfighters  and  the  decisions  required  to  meet 
those  goals,  the  nature  of  the  operating  environment  is  a  major  component  to  operational  success. 
Because  multiple  factors  affect  the  human  component  of  C2  performance,  a  new  methodology 
employing  the  best  practices  from  current  cognitive  engineering  processes  is  required.  This 
method  brings  together  analyses  and  techniques  from  Scenario  Based  Design,  Situation 
Awareness  Analysis,  Cognitive  Task  Analysis,  Team  Cognitive  Task  Analysis,  Cognitive  Work 
Analysis,  Use  Cases  and  Storyboarding. 

EDAM  (see  FIGURE  2)  begins  with  the  fundamentals:  scenario  design/articulation  and  an  initial 
work  domain  creation.  These  fundamentals  lead  into  a  knowledge  elicitation  phase  of  mission 
functions.  The  knowledge  elicitation  is  conducted  on  two  parallel  paths  addressing  cognitive 
performance,  one  focusing  on  decisions  and  the  other  on  the  work  environment.  Knowledge 
elicitation  involves  various  activities,  from  attending  function  related  courses  and  visiting  training 
sites,  command  centers,  and  other  C2  intensive  sites  to  interviewing  subject  matter  experts 
(SMEs)  with  semi-structured  and  structured  techniques.  The  two  paths  converge  on  decision  and 
work  environment  analysis,  which  involves  producing  various  representations  of  the  knowledge 
elicited.  The  analysis  and  the  resulting  representations  lead  to  decision  support  system  (DSS) 


Minor  Change  - 

C2  structure  left  intact 
C2  systems  slightly  altered 
New  user  front-end 

Evolutionary  Change  - 

C2  structure  left  intact 
C2  systems  redesigned 

New  user  front-end 

Revolutionary  Change  - 

C2  structure  redesigned 
C2  systems  redesigned 

New  user  front-end 
New  work  structure 


FIGURE  1  Change  Continuum 


design  concepts.  The  DSS  design  concepts  can  now  be  combined  with  technology  assessments  to 
create  a  total  system  design  concept  and,  ultimately,  a  demonstrable  prototype.  Note  that 
although  these  steps  are  listed  as  a  sequence,  concurrent  and  iterative  work  will  take  place 
throughout  the  methodology.  Furthermore,  the  methodology  is  best  applied  by  a  multi¬ 
disciplinary  team  that  provides  systems,  human  performance,  software,  hardware,  and 
operational/domain  views  into  the  design  effort. 

Throughout  the  design  effort,  human  performance  assessment  is  conducted  to  the  degree  possible. 
Initially,  assessment  occurs  through  paper-based  modeling  and  simulation  to  understand 
workload,  support  function  allocation  and  organization  design  decisions.  Later  in  the  design 
process,  human-in-the-loop  assessments  of  design  concepts  and  the  prototyped  demonstrator 
provides  performance  evaluations.  Throughout  the  design  and  development  process,  SME 
participation  is  critical.  Domain  knowledgeable  individuals  assist  in  developing  the  scenarios 
that  are  used  for  interviews  with  current  warfighters  to  elicit  decision  and  work  environment 
requirements,  and  SMEs  participate  in  design  evaluations. 

Familiarization  of  the  existing  domain  and  job  environment  is  extremely  important  for  all  team 
members.  In  particular,  EDAM  team  members  review  or  study  relevant  material,  such  as  work 
process  documentation  (e.g.,  Tactics,  Techniques  and  Procedures  (TTPs),  Doctrine),  training 
packages,  and  operational  requirements;  participate  in  training;  and  observe  operations  and 
exercises. 

The  following  sections  describe  in  detail  each  of  the  steps  discussed  above.  In  addition,  potential 
products  from  each  step  are  presented. 

EDAM  COMPONENTS 

Scenarios 

C2  operations  involve  many  complex  functions.  To  fully  understand  the  effects/goals  that 
support  mission  accomplishment,  C2  must  be  assessed  in  context.  Since  revolutionary  designs  do 
not  offer  an  existing  system  and  work  organization  to  analyze,  EDAM  calls  for  developing  one  or 
more  operation  scenarios  to  provide  that  context.  SMEs  aid  in  the  development  and  validation  of 
the  scenarios.  The  EDAM  process  identifies  scenario  specificity  requirements,  and  domain 
experts  help  to  ensure  realism.  The  scenarios,  depicted  in  textual  and  graphic  format,  are  used 
during  the  knowledge  elicitation  phase  to  structure  the  interview  process. 

In  addition  to  initial  knowledge  elicitation,  the  scenarios  are  used  at  all  stages  of  the  EDAM 
process  to  provide  context.  As  such,  it  is  important  that  warfighter  SME  interviews  are  structured 
to  enhance  and  update  the  scenarios’  descriptions,  logical  essentials  (e.g.,  information 
requirements),  and  process  steps  to  reflect  reality  and  the  new  C2  environment. 

EDAM’s  use  of  scenarios  is  based  on  the  cognitive  engineering  approach  scenario-based  design 
(McGraw  and  Harbison,  1997).  The  primary  benefits  of  scenario-based  engineering  -  to  bring 
developers  and  process  engineers  together  (in  our  case  cognitive  process  engineers)  in  a 
structured  environment  -  are  leveraged  in  the  EDAM  process.  The  focus  on  scenarios  helps  to 
ensure  that  all  of  those  involved  in  the  design  development  are  working  within  a  realistic  context. 


SME  Participation 


abstraction  hierarchy  can  also  be  decomposed  into  various  levels  of  subsystems  called  a 
decomposition.  The  decomposition  is  generally  represented  from  left  to  right  with  the  whole 
system  at  the  left  and  subsequent  subsystems  to  the  right. 

The  control  task  analysis  is  generally 
represented  with  multiple  decision  ladders; 
a  generic  decision  ladder  is  shown  below 
(see  FIGURE  4).  A  decision  ladder  is  not 
constrained  by  who  (either  human  operator 
or  machine  operator)  is  controlling  or 
interacting  with  the  domain  or  system. 

Eventually,  the  results  of  the  work  domain 
and  control  task  analyses  leads  to  the 
development  of  a  function  allocation, 
initial  user  profiles,  and  machine 
capabilities.  For  future  C2  systems, 
manning  organizations  are  difficult  to  base 
on  an  evolution  of  current  manning  structures.  Instead,  the  EDAM  approach  identifies  the 
required  functionality  and  the  requisite  knowledge,  skills,  and  abilities  (KSAs)  required  to 
perform  those  functions.  Once  the  required  KSAs  are  identified,  functions  can  be  aggregated  into 
roles;  these  roles  will  provide  a  framework  for  developing  the  initial  user  profiles.  Throughout 
the  EDAM  process,  the  initial  functional  allocation  is  assessed  for  workload  impacts  and 
decision/task  reassignments,  and  user  profiles  are  refined. 

Decision  Knowledge  Elicitation 

To  refine  and  authenticate  the  information  gleaned 
from  the  available  documentation,  interviews  with 
SMEs  are  performed.  The  decision  knowledge 
elicitation  focuses  on  the  goals  that  support  the 
mission,  the  decisions  necessary  to  achieve  those 
goals,  and  the  information  requirements  and 
strategies  used  for  each  decision,  based  on  a  Goal- 
Directed  Task  Analysis  (GDTA)  (Endsley,  2003). 
During  semi -structured  interviews,  the  EDAM 
team  walks  SMEs  through  the  developed 
scenarios.  (Note:  While  individual  interviews  are 
ideal,  schedule  and  resource  limitations  may 
require  group  interviews.  In  this  case,  the 
elicitation  activity  resembles  more  of  a 
workshop/roundtable  approach  than  an  interview). 

This  analysis  captures  user  goals  and  corresponding 
FIGURE  4  Control  Task  Analysis  Result  -  decisions,  yielding  situational  assessment  and 
Decision  Ladder  actionable  information  requirements.  This 

information  is  represented  as  shown  in  FIGURE  5.  In 
addition  to  a  combined  list  of  decisions  made,  actionable  information  needed  to  make  such  decisions, 
and  the  interactions  and  information  flow  among  decisions  emerge.  This  information/decision  flow 
data  is  best  represented  by  flow  charts. 
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FIGURE  3  Work  Domain  Analysis  Result  - 
Abstraction  Hierarchy 


To  provide  more  detailed 
decision  analysis,  the 
EDAM  team  also  conducts  a 
Cognitive  Task  Analysis 
(CTA).  Several  methods  for 
conducting  CTAs  exist. 

One  method,  based  on  the 
work  of  Gary  Klein 
(Crandall,  B.,  Klein,  G., 
Militello,  L.,  &  Wolf,  S„ 
1994),  is  the  knowledge 
audit.  A  knowledge  audit 
consists  of  interviews  with 
SMEs  to  mine  their 
expertise.  The  SME  is 
queried  on  the  actions  he/she 
would  take  at  each  decision 
point.  Complex  decisions 
are  examined  in  more  depth 
FIGURE  5  Goal-Directed  Task  Analysis  Output  t0  focus  on  t^c  factors  that 

contribute  to  expertise  (e.g., 

cues  and  strategies  used).  This  information  can  aid  in  designing  systems  that  help  novice 
operators  perform  more  like  experts  despite  their  lack  of  experience.  It  can  also  provide 
information  regarding  personnel  and  training  needs. 

As  there  is  no  existing  system  or  domain,  scenario-based  techniques  are  applied;  the  tactically- 
realistic  scenarios  developed  and  validated  previously  are  used  to  walk-through  the  questions  to 
elicit  decision  making  requirements.  The  table  below  provides  an  example  used  to  elicit  high 
level  decisions  and  accompanying  requirements  for  an  Air  Force  Combined  Air  and  Space 
Operations  Center  time  sensitive  targeting  system. 

Work  Environment  Knowledge  Elicitation 

Conducted  concurrently  with  the  GDTA  and  CTA  described  above,  the  Work  Environment 
Knowledge  Elicitation  creates  a  representation  of  the  physical  work  environment  and 
organizational  structure.  In  addition  to  the  actual  tasks  and  decisions,  the  environment  and 
organization  also  impacts  the  work  that  is  accomplished.  This  information  is  collected  through 
field  observation  and  ethnographic  studies,  which  involve  observing  actual  C2  operations, 
exercises,  and/or  training  to  understand  and  accurately  document  the  current  operational  process, 
strategies,  and  tools.  The  following  activities  are  conducted  to  ensure  useful  and  complete 
knowledge  is  gained  from  the  observations: 

a.  Review  doctrine  prior  to  exercise  with  C2  representatives  and  trainers  to  identify  holes  in 
doctrine.  Observe  exercise(s)  and  document  divergence  from  doctrine  and  established 
processes. 

b.  Collect  data  passed  to/from  operator/supervisor,  observe  use  of  data  and  by  whom,  observe 
method/tool  used  to  pass  and  process  data,  document  cognitive  decision-making  process  used 
by  operator/supervisor  (may  require  post-hoc  discussions). 

c.  Conduct  post  exercise  interviews  with  participants,  including:  conducting  knowledge  audit; 
documenting  rationale  for  divergence,  difficult  aspects/events  in  exercise,  paths  of 
communication  used  and  alternatives  not  employed,  heuristics  and  experiential  knowledge 


and  how  that  is  applied  to  situation  at  hand;  reviewing  errors;  collecting  strategies,  tools, 
communication  channels,  communication  flow  not  observed  but  used  in  exercise  (will  require 
post-hoc  discussions). 

d.  Assess  usability  issues  associated  with  systems  used  in  exercise  (questionnaire,  subjective 
evaluations). 

e.  Determine  task  frequencies  and  durations,  task  fragmentation/interruptions,  fatigue  and 
stress,  and  performance  requirements. 


TABLE  1  Example  questions  for  decision  knowledge  elicitation 


Probe 

Questions  to  ask  if  SME  is  having  a  hard  time 
identifying  decisions/information  requirements 

Goals 

•  What  are  your  specific  goals  at  this  time? 

•  How  did  you  prioritize  these  goals?  Are  there 
conflicts? 

•  How  does  the  outcome  of  a  particular  goal 
influence  the  success  of  your  prioritization/re¬ 
prioritization  of  other  goals? 

•  Does  this  scenario  fit  a  standard  or  typical 
scenario? 

•  Did  this  scenario  remind  you  of  any  previous  case 
or  experience? 

Decisions 

•  In  this  event  in  the  scenario,  what  are  the  key 
decisions  you  are  making? 

•  What  decisions  would  you  actually  make? 

•  What  would  cause  you  to  make  a  decision  at  the 
time  the  decision  was  made? 

•  What  decisions  would  be  considered  but  would 
be  deferred  and  why? 

•  Would  any  decisions  be  made  in  collaboration 
with  other  CAOC  staff? 

•  Would  any  decisions  made  require  review  and 
approval  from  other  CAOC/TCT  staff? 

Information  Requirements 

•  In  this  situation,  how  would  you  go  about  making 
the  required  decisions,  what  would  cue  you? 

•  What  information  would  you  seek? 

•  Can  you  get  this  information?  Is  it  currently 
available? 

•  Who  would  you  interact  with? 

•  What  ambiguities  would  you  try  to  resolve? 

•  How  do  you  communicate  with  external  teams 
(i.e.,  SOF),  particularly  in  event  3? 

•  How  do  you  combine  information  (use  specifics) 
to  aid  in  decision  making? 

Errors 

•  What  types  of  errors  are  likely  at  this  point?  In  this 
decision? 

•  What  makes  this  difficult? 

Situation  Awareness  -  ability  to  respond 

•  How  do  you  maintain  SA,  what  are  you  looking 
for,  where  does  it  come  from  (who  are  you  talking 
to,  what  displays/systems  do  you  use)? 

•  Who  are  you  talking  to,  what  displays/systems  do 
you  use? 

Response 

•  What  is  the  effect  of  the  decision?  To  whom  are 
you  providing  information  and  action  cues? 

•  Who  needs  to  know  your  decisions,  information 
generated? 

Decision  and  Work  Environment  Analysis 

Throughout,  and  after  the  completion  of,  the  knowledge  elicitation  phase,  the  data  and 
information  collected  are  analyzed  and  represented  in  various  ways.  When  designing 
revolutionary  C2  systems,  the  EDAM  team  focuses  the  representations  on  high  level  abstract 
functions,  goals  and  decisions  and  not  on  current  practices  and  policies.  A  representation  that 
integrates  several  types  of  information  into  one  diagram  is  an  operational  sequence  and  activity 


diagram,  which  provides  a  graphical  process  flow  divided  across  the  different  human  and 
machine  roles  over  time.  An  example  is  shown  in  FIGURE  7. 


FIGURE  7  Example  of  Operational  Sequence  and  Activity  Diagram 


In  addition,  a  function  allocation  matrix  is  produced.  This  matrix  matches  the  domain  functions 
to  various  human  and  machine  roles.  An  example  is  shown  in  FIGURE  8.  Both  of  these 
representations  accompany  all  the  representations  developed  in  previous  stages  to  show  a  clear 
picture  of  the  many  aspects  that  affect  work  completed  within  the  C2  system. 

Decision  Support  Systems  Design 
Concept 


This  phase  of  EDAM  specifies  and 
develops  operational  aspects  of  the 
design  -  what  the  system  does,  and  how 
the  user  interacts  with  it.  In  addition,  it 
identifies  system  states  that  need  to  be 
revealed  to  the  user,  makes  clear  what 
user  actions  are  available,  and  generates 
system  responses  that  need  to  be  presented  through  a  graphical  user  interface  (GUI). 

The  knowledge  from  the  previously  described  knowledge  representations  is  now  transferred  into 
formats  that  the  software  development  team  can  use  to  produce  a  prototype  demonstration 
system.  Translating  the  results  of  the  Decision  and  Work  Environment  Analysis  into  a  form  that 
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FIGURE  8  Example  of  Generic  Function  Allocation 
Matrix 


software  developers  can  use  has  been  a  weak  link  in  traditional  cognitive  engineering  efforts.  To 
bridge  this  gap,  the  EDAM  embraces  unified  modeling  language  (UML)  techniques  to  create  use 
cases  and  activity  diagrams.  Standard  development  representations  are  also  created,  including 
annotated  story  boards  (paper  prototypes  which  demonstrate  the  graphics,  interactions,  and 
navigation  of  the  proposed  GUI). 

Employing  these  software  development-focused  techniques  eases  the  transition  from  the  initial 
EDAM  products  to  a  prototype  that  software  developers  can  implement.  Moreover,  this  process 
is  not  isolated  from  the  developers  and  systems  engineers  and  is  tied  closely  to  the  total  System 
Design  Concept  phase  described  below.  All  previously  described  knowledge  representations  are 
also  available  to  the  developers  and  systems  engineers.  Furthermore,  representatives  from  these 
teams  are  encouraged  to  participate  in  all  interactions  with  C2  representatives  and  SMEs. 

Technology  Trade-Off  Study 

The  software  development  team  leads  technology  trade-off  studies.  The  technology  trade-off 
study  determines  the  best  hardware  and  software  configuration  for  developing  and  presenting  a 
prototype  demonstrator.  Throughout  the  study,  the  EDAM  team  assists  by  providing  expertise  in 
human  factors,  usability,  and  supportability  concerning  warfighter  C2  decisions-making  issues. 

System  Design  Concept 

The  total  system  design  concept  defines  the  system’s  physical  work  environment,  organizational 
structure,  policies,  and  hardware.  When  revolutionary  systems  are  developed  using  EDAM,  it  is 
anticipated  that  changes  and  reduction  in  the  organizational  structure  will  also  occur,  which,  in 
turn,  may  require  changes  in  operational  policies.  In  addition,  the  hardware  used  and  the  work 
environment  are  likely  to  differ  dramatically  from  what  is  currently  used. 

To  better  understand  the  organizational  structure  and  role  changes  that  will  best  meet  the  needs  of 
the  new  work  domain  design,  the  EDAM  team  conducts  workload  analyses  using  modeling  and 
simulation  to  examine  different  task  configurations  among  the  personnel.  This  modeling  and 
simulation  may  be  conducted  in  Micro  Saint  Sharp  or  other  discrete  event  simulation  tools. 

Furthermore,  anthropometric  considerations  need  to  be  reviewed.  The  design  should  utilize 
hardware  that  adapts  adequately  to  varying  personnel  sizes  and  builds.  Recommendations  are 
documented  and  incorporated  into  overall  system,  GUI,  and  user  interface  designs. 

As  described  previously,  the  System  Design  Concept  phase  occurs  in  conjunction  with  the 
Decision  Support  System  Design  Concept  phase,  and  the  new  operational  policies  are 
incorporated  into  use  cases,  activity  diagrams,  and  annotated  storyboards.  In  addition,  the  EDAM 
team  produces  graphics  (e.g.,  layouts,  hardware  designs,  images,  etc.)  of  the  hardware  and  work 
environment  configuration,  including  diagrams  of  the  information  flow  and  magnitude  into, 
through,  and  out  of  the  C2  system. 

Prototype 

Various  methods  of  prototyping  exist,  ranging  from  paper  sketches  to  full  working  systems. 
EDAM  encompasses  the  earliest  types  of  prototypes:  storyboarding  and  requirements  animation. 
These  prototypes  are  used  to  communicate  design  guidelines  to  the  software  development  team  as 
well  as  to  support  human  performance  assessments,  cognitive  walkthroughs,  and  usability  tests 


with  C2  representatives.  Storyboarding  is  normally  a  paper  prototype  while  requirements 
animation  is  a  dynamic  representation  but  not  a  working  system.  The  software  development  team 
continues  to  refine  the  prototypes  by  developing  higher  fidelity  prototypes,  including  rapid 
prototypes  which  can  be  used  to  determine  the  adequacy  of  the  design  but  do  not  themselves 
evolve  into  a  final  solution,  incremental  prototyping  which  builds  the  system  in  phases  adding 
features  over  time,  and  evolutionary  prototyping  which  uses  evaluation  and  iteration  to  refine  the 
prototype  into  a  final  demonstration  system.  The  EDAM  team  supports  the  software 
development  team’ s  prototype  development  by  providing  expertise  on  how  to  meet  the  human 
decision  making  requirements  within  the  system. 

Human  Performance  Assessment 

EDAM  incorporates  human  performance  assessments  throughout  the  design  process  to  validate 
analyses  conducted,  requirements  derived,  and  designs  developed.  This  performance  assessment 
is  based  on  modeling  and  simulation  that  evolves  from  sparse  scenarios  through  detailed  paper 
prototypes.  The  human  performance  assessment  continues  through  the  software  development 
team’s  prototypes  as  well. 

Metrics  that  are  appropriate  for  a  particular  system  should  be  developed  as  the  design  is  being 
developed,  but  a  short  list  of  generic  metrics  includes:  successful  completion  of  tasks,  time  to 
complete  a  specified  task  by  a  specified  user,  percent  of  task  completed,  time  spent  on  errors, 
number  of  errors  per  task,  number  of  commands  used  to  accomplish  task,  frequency  of  and  time 
spent  on  help  or  documentation  use,  number  of  runs  of  success  and  of  failure,  number  of  available 
commands  not  invoked,  number  of  regressive  behaviors,  number  of  time  users  need  to  work 
around  a  problem,  and  subjective  user  evaluations.  These  metrics  are  associated  with 
performance  goals  and  are  evaluated  throughout  the  EDAM  and  development  processes. 

CONCLUSION 

Warfighters  using  today’s  C2  systems  make  numerous  critical,  time  constrained  decisions 
requiring  a  large  amount  of  information.  However,  large  screens  inundated  with  data,  legacy 
systems  with  limited  interoperability,  and  poorly  designed  operator-machine  interfaces  force  the 
warfighter  to  spend  more  time  interacting  with  systems,  analyzing  information,  and  developing 
situational  awareness  than  making  critical  decisions.  Effective  military  operations  are  enabled  by 
a  system  centered  around  human  perception,  understanding,  abilities,  and  decision  making, 
providing  actionable  information  at  the  appropriate  time,  situation  awareness  through  the  display 
of  knowledge  (not  data),  and  natural,  intuitive  interaction  technologies.  In  short,  future 
warfighters  need  the  right  information,  at  the  right  time,  in  the  right  modality. 

Revolutionary  C2  systems  that  provide  decision  support  tools  and  a  work  environment  that 
support  rapid,  accurate  decision-making  benefit  from  the  application  of  warfighter-centered 
design  approaches.  Current  cognitive  engineering  methods  don’t  have  the  breadth  required  to 
adequately  address  the  human  performance  considerations  from  organizational  and  work 
environment  to  the  low-level  detailed  aspects  of  complex  decision-making  that  can  be  codified 
into  automation  and  decision  support.  Applying  a  hybrid  approach,  like  EDAM,  can  help  ensure 
that  new  C2  systems  meet  user  performance  requirements.  Consideration  of  the  operating 
environment,  collaboration  between  humans  and  machines,  user  goals,  operator  decision  methods 
and  styles,  battle  rhythm  dynamics,  multi-level  security,  cognitive  strengths  and  limitations  and 
physiological  factors,  such  as  stress  and  fatigue,  all  of  which  are  addressed  and  incorporated  into 
design  solutions. 
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Continuum  of  Char 


Minor  Change 

C2  structure  left  intact 
C2  systems  slightly  altered 
New  user  front-end 

Evolutionary  Change 

C2  structure  left  intact 
C2  systems  redesigned 
New  user  front-end 
Changes  to  back-end 

Revolutionary  Change 

C2  structure  redesigned 
C2  systems  redesigned 
New  user  front-end 
New  back-end 
New  work  structure 


Minor  Change 
Many  cognitive 
systems  engineering 
methods  available 

Evolutionary  Change 

Some  cognitive 
systems  engineering 
methods  available 


Revolutionary  Change 
Very  few  cognitive 
systems  engineering 
methods  available 


Effects-based  Decision  Analysis 
Methodology  (EDAM) 


•  Designed  for  revolutionary  change 

•  Based  on  cognitive  engineering  methodologies 

-  Scenario  Based  Design 

-  Cognitive  Work  Analysis 

-  Situation  Awareness  Analysis 

-  Cognitive  Task  Analysis 

-  Team  Cognitive  Task  Analysis 

-  Use  Cases 

-  Storyboarding 


Multiple  methods  to  meet  complete  system  design 


Scenario  Design 


•  Provide  context 

•  Textual  and/or 
graphical 

•  Used  for  structure 
throughout  EDAM 
from  knowledge 
elicitation  through 
human 
performance 
evaluations 


Scenario  Based  Design 


Work  Domain  Analysis 


•  Defines  a  work  domain’s 
goals  and  constraints 

•  Often  portrayed  by  a 
combined  abstraction 
hierarchy  and  system 
decomposition 

•  Updated  and  refined 
throughout  EDAM  to 
reflect  current  design  and 
ensure  that  design 
concepts  meet  the  high- 
level  goals 


Whole 

System 

Subsystem 

A 

Subsystem 

B 

Functional 

Purpose/Goal 

Abstract 

Function 

Generalized 

Function 

Physical 

Function 

Physical  Form 

Cognitive  Work  Analysis 


Control  Task  Analysis 


Describes  means 
of  controlling  or 
interacting  with 
work  domain 

Independent  of 
agent  (human  or 
machine) 

Can  be  used  to 
aid  in  the 
completion  of  a 
function/task 
allocation  matrix 


Cognitive  Work  Analysis 


©Decision  Knowledge 
Elicitation 


Goals 

What  are  your  specific  goals  at  this  time? 

How  do  you  prioritize  these  goals?  Are  there  conflicts? 

How  does  the  outcome  of  a  particular  goal  influence  the 
success  of  your  prioritization/reprioritization  of  the 
_ other  goals? _ 

Decisions 

In  this  event  in  the  scenario,  what  are  the  key  decisions 
_ you  are  making? _ 

Information  Requirements 

In  this  situation,  how  would  you  go  about  making  the 
required  decisions,  what  would  cue  you? 

What  information  would  you  seek? 

Can  you  get  this  information?  Is  it  currently  available? 
Who  would  interact  with  you? 

What  ambiguities  would  you  try  to  resolve? 

Errors 

What  type  of  errors  are  likely  at  this  point?  In  this 
decision? 

_ What  makes  this  difficult? _ 

Situation  Awareness  -  ability  to  respond 

How  do  you  maintain  SA,  what  are  you  looking  for, 
where  does  it  come  from? 

Response 

What  is  the  effect  of  the  decision?  To  whom  are  you 
providing  information  and  action  cues? 

Who  needs  to  know  your  decisions,  information 
generated? 


Structured  by  scenarios 


Focused  on  goals  and 
decisions  as  opposed  to 
current  methods  or 
systems 

Interviews  with  Subject 
Matter  Experts,  in  groups 
and  individually 


Cognitive  Task  Analysis 


Decision  Knowledge 
Elicitation 


•  Goal  Directed 
Task  Analysis 

•  Decomposes 
to  increasing 
levels  of  detail 


Derives  requirements  for  decision  support  system  design 


Situation  Awareness  Analysis 


Work  Environment 
Knowledge  Elicitation 


•  Field  observations  and  ethnographic  studies  of 
C2  operations,  exercises,  and/or  training 

•  Focus  on  work  in  context  -  environment,  layout 
of  physical  space,  equipment,  formal  and 
informal  social  organization,  communication 

•  Possible  steps  to  follow  for  timeliness: 

-  Review  doctrine  prior  to  exercise  or  observation 

-  Observe  work  in  actual  work  environment 

-  Conduct  post-observation  interviews  with  observed 
subjects 


Derives  requirements  for  complete  system  design 


Team  Cognitive  Task  Analysis 


Decision  and  Work 
Environment  Analysis 


Knowledge  Organization  and  Representation 

Activity  Diagram 


Function  Allocation  Matrix 
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Abstracts  knowledge  gained  in  field  observations 


Technology  Trade-Off 

- J*  K 

* - *  ^ 

Study 

•  Led  by  software/hardware  development  team 

-  Consider  future  and  near-future  technologies 

•  Input  from  EDAM  team 

-  human  roles 

-  human  factors 

-  usability 

-  supportability 


Selecting  the  optimal  technology  (cost/performance 

trade)  to  support  decision  making 


Decision  Support  Systems 
Design  Concept 


•  Use  Cases  in  UML 

-  Operational  aspects  of 
the  design 

•  what  the  system  does 

•  how  the  human 
interacts  with  the 
system 


•  Storyboards  and 
Paper  Prototypes 

-  Graphical  User 
Interface  concepts 

•  information  architecture 

•  interactions 

•  navigation 

•  graphics 


Easing  the  transition  to  software  developers 


Use  Cases  and  Storyboarding 


System  Design  Concept 


Concurrent  with  decision  support  system  design 
Complete  system  design 

-  physical  work  environment 

-  organizational  structure 

-  policies  and  procedures 

-  software/hardware 

Supported  by  workload  modeling  and  simulation 


Supports  all  aspects  of  decision  making 


Prototype 


•  Levels  of  prototyping 

-  Storyboards  and  requirements  animation  (EDAM 
team) 

-  Rapid,  incremental,  and  evolutionary  (S/HD  team) 

•  Uses 

-  Translate  requirements  from  EDAM  to  SD  team 

-  Human  performance  assessments 

•  cognitive  walkthroughs 

•  usability  testing 


Enables  iterative  evaluate/testing  of  system  design 


Human  Performance 
Assessment 


•  Performed  frequently  throughout  EDAM 

-  Identify  early  as  part  of  requirements  analysis,  validate  metrics 
using  prototypes  and  modeling  &  simulation 

-  sparse  scenarios  to  detailed  paper  prototypes  to  software 
development  prototypes 

•  Metrics  developed  as  appropriate  to  the  system 

-  successful  and  timely  completion  of  tasks 

-  time  spent  on  errors,  number  of  errors 

-  number  of  actions  used  to  complete  task 

-  number  of  regressive  behaviors 

-  subjective  user  evaluations 


Provides  technical  performance  measures  for 
the  human  element  of  the  system 


Conclusion 


Minor  Change 


Evolutionary 

Change 


Revolutionary 

Change 


C2  Structure  intact  and  Systems  Altered 

New  user  front-end 

Many  Cognitive  Engineering  Methods 


C2  Structure  intact  and  Systems  Redesigned 

New  user  front-end 

Changes  to  back-end 

Some  Cognitive  Engineering  Methods 

C2  Structure  and  System  Redesigned 
New  user  front-end 
New  back-end 
New  work  structure 

Very  few  Cognitive  Engineering  Methods 


EDAM  address  the  human  element  in  C2  design 
and  fills  the  gap  in  revolutionary  change 


QUESTIONS? 


Decision  Knowledge  Elicitation 


Probe 

Things  to  consider  asking  if  SME  is  having  hard  time  identify 
decisions/information  requirements 

Goals 

what  were  your  specific  goals  at  the  time? 

does  this  scenario  fit  a  standard  or  typical  scenario? 

How  did  you  prioritize  these  goals?  Are  there  conflicts? 

how  does  the  outcome  of  a  particular  goal  influence  the  success  of  or 
your  prioritization/re-prioritization  of  other  goals? 

decisions 

in  this  event  in  the  scenario  what  are  the  key  decisions  you  are  making? 

did  this  scenario  remind  you  of  any  previous  case  or  experience? 

what  decisions  would  you  actually  make? 

what  would  cause  you  to  make  a  decision  at  the  time  it  was  made 

what  decisions  would  be  considered  but  deferred/why? 

would  any  decisions  be  made  in  collaboration  with  other  staff? 

would  any  decisions  made  require  review  and  approval 

information  requirements 

in  this  situation  how  would  you  go  about  making  the  required  decision, 
what  would  cue  you, 

how  do  you  communicate  with  external  teams  (i.e.  SOF),  particularly  in 

Event  3? 

what  informaiton  would  you  seek, 

how  do  you  combine  information  to  aid  in  decision  making? 

can  you  get  this  information?  Is  it  available? 

who  would  you  interact  with, 

what  ambiguities  would  you  try  to  resolve? 

errors 

what  type  of  errors  are  likely  at  this  point?  In  this  decision? 

what  makes  this  difficult? 

situation  awareness  -  ability  to  respond 

how  do  you  maintain  SA,  what  are  you  looking  for,  where  does  it  come 
from 

who  are  you  talking  to,  what  displays/systems  do  you  use? 

response 

what  the  effect  of  the  decision,  who  are  your  providing  information,  what 
are  the  action  cues  to  respond 

who  needs  to  know  your  decisions,  information  generated 

